ALLEGHENY COUNTY DEPARTMENT OF HUMAN SERVICES 

Using Predictive Modeling 
to Improve Outcomes 

For Children in Allegheny County 


Training for Child Welfare 
Professionals 



Agenda 


• Objectives 

• Overview of Predictive Analytics/Predictive Risk Modelling 

• Application of Predictive Analytics in Allegheny County 

• Policy/Practice 

• Case scenarios 

• Process changes 

• KIDS Changes/Design 

• Questions 
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Objectives 


By the end of this training, participants will be able to... 

• Understand the basics of Predictive Analytics/Predictive Risk Modeling 

• Describe how the Family Screening Score will be generated in KIDS 

• Understand what the Family Screening Score means 

• Explain how the Family Screening Score is to be included in making screening decisions and the policy 
for its use 

• Understand the addition of dynamic text from Client View to a referral in KIDS 


Allegheny County Department of Human Services 
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I What is Predictive 
Risk Modeling? 




Today: Using Integrated Data 
to Inform Decision-Making 


• In Allegheny County, rich data are available to 
workers and supervisors to help inform 
decisions at call screening, but 

- No standardized protocols for using these data to 
inform screening decisions 

- No method for systematically weighting this 
information in an equitable manner across all referrals 

- No understanding of what information is correlated / 
predicts future adverse outcomes for children 


• Is there a way to address any of these concerns? 
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What is Predictive Risk Modeling? 


• Use data already collected about each child to evaluate the possibility of future 
adverse outcomes 

• Why consider predictive risk modeling? 

- Wider availability of high quality information about service interactions that can inform our 
understanding of risk and protective factors 

- Advances in technology and analytic capabilities- we can use these data in real time rather than 
retrospective analyses 

- Difficulty in weighting complex factors (and time is scarce!) 


Allegheny County Department of Human Services 
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I How is Predictive Analytics 
Being Applied in Allegheny 
County? 




Developing a Screening Score 


• We developed a partnership with a research 
team based in New Zealand 

• We shared Allegheny County data with the 
research team to analyze for factors that would 
be predictive of future maltreatment 

• Call screening was selected as the decision point 
at which a model would be developed and used 

• The team developed a model that includes more 
than 100 pieces of information 

• This screening score is comprised of the 
likelihood of re-referral, given screen-out and the 
likelihood of placement in foster care, given 
screen-in 
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Running a Screening Score 


• The tool is called the Allegheny Family Screening Tool 

• The score that will be produced is called the Family Screening Score 

• The score will be run in the KIDS system which will harvest data from the data warehouse* 

• At each call, a screening score will be run for each child associated with an allegation of 
maltreatment 

• Only one score will be displayed to the screener- the highest score for the highest child on that 
referral 

• The score only applies at call screening 

• The score will only be viewed by call screeners and call screening supervisors 


Allegheny County Department of Human Services 


* Only if on MCI_ID is successfully established 
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Developing a Screening Score 


• The screening score is from 1 to 20 (with a subset of referrals being mandatory screen ins) 

• The higher the score, the higher the chance of the future event 

(e.g., abuse, placement, re-referral) according to the data 



4 5 6 7 


i 

a 

1 in 20 

calls receive 

a score of 1 



(bottom 5 % of risk) 


13 14 15 16 17 18 19 20 


i 

i 

i 

1 in 20 

calls receive 

a score of 20 

(top 5 % of risk) 


Allegheny County Department of Human Services 


10 
















Okay. 


But will it work? Will it actually give me 
a score that reflect this family's likelihood of 
future maltreatment? 

To test that, we scored thousands of 
historical maltreatment calls and then 
followed the children in subsequent 
referrals to see how often the model was 
correct... 



Developing the model 


r -®53% ( 12 , 894 ) 

! Screened Out 



24,188 

Index Referrals 


--©47% ( 11 , 214 ) 

Screened In 


f© 35% ( 4 , 467 ) 

Re-referred within a year 
j 

-•©65% ( 8 , 418 ) 

No Re-referrals 


f -©24% ( 2 , 681 ) 

Re-referred within a year 

•i 


I 

[ --©76% ( 8 , 533 ) 

No Re-referrals 
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The 



1 in 10 children 

with a score of 1 were re-referred 

within two years of the call. 
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: Re-Referrals 


9 in 10 children 

with a score of 20 were re-referred 

within two years of the call. 
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The Results: 

Out-of-Home Placements 





1 in 100 children. 

who received a score of 1 were placed 
out-of-home within 2 years of the call 
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The Results: 

Out-of-Home Placements 





1 in 2 children 


who received 

a score of 20 were placed 
out-of-home within 
2 years of the call 
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Under current practice: 


27% of highest risk cases 
were screened out — 

of these, 1 in 3 are re-referred 
and placed within 2 years of the 
initial screened out call 

48% of lowest risk cases 
were screened in — 

only 1.4% of those 
are placed within 2 years. 
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Approaches to Assessing Risk 



PA Risk Assessment 

Family Screening Score 

Definition 

Likelihood of future maltreatment 

Likelihood of future re-referral or placement 

Based on 

Clinical judgement of worker 

Data driven 

Timeframe 

Future 

Within two years of call 

Type assessed 

Acute and chronic 

Chronic only 

Considers 

All aspects of the family life relevant to 
future maltreatment (allegations, family 
stressors, prior history) 

All factors found to be predictive through 
modeling of all Data Warehouse variables for 
CYF referrals 2010-2013 

Includes 

15 factors 

Over 100 factors 

Applies 

At all critical child welfare decision points 
and when new information arises 

Only at call screening decision 

Output 

Overall Severity: seriousness of what has 
happened 

Overall Risk: likelihood of future harm 

Highest score for highest child for likelihood of 
either re-referral or placement 





























I What is the Policy and 
Process for 
Using the Family 
Screening Score? 



Policy 


(DRAFT) 



The Allegheny County Office of Children, Youth, and 
Families (OCYF) Call Screening Department will 
employ the Allegheny Family Screening Tool as one of 
the tools used in determining a screening decision for 

all incoming referrals. 


Process change 


• Screeners will still... 

- Take call and record information 

- Generate client list 

- Review previous CYF history, Client View, other sources of information 

• Screeners will now... 

- Complete a risk and safety rating 

- Review the Family Screening Score 

- Offer a recommendation 

- Send the referral for approval to the supervisor 
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Process Change at Call Screening 


New Call Screening Business Process 

Incorporating Family Screening Score utilization and Other 8/1/16 KIDS changes 


Call Screener Findings 





Field Screening 

(if necessary) 

If Call Screening Supervisor 

determines a field screening is 
necessary, they redirect referral to 


C 


Disposition 
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Call Screening 
Supervisor receives 
referral and 
recommendation, and 
determines next steps 


If they have not 
already done so in 
determining need for 
a Field Screening, Call 
Screening Supervisor 

must generate a 
Family Screening 

Score . 


f Field Screening Findings 


Field Screening Unit 

completes a minimum of 
a Contact and a Safety 
Assessment and enters a 

Field Screening 
Recommendation into 
KIDS, which is submitted 
back to the Call 
Screening Supervisor. 



Mandatory Screen-Ins: 

If Family Screening Score is high enough to trigger a 
"Mandatory Screen-In,“ the Call Screening Supervisor 
must include a mandatory override justification in 
order to Screen Out. Instances that may utilize 
override include: 

• Referrals from outside county 

• Referrals created erroneously 

• Referrals wthout any allegation of abuse (e.g.. a 
client seeking to be connected to community 
resources) 

• Courtesy visits for other jurisdictions 

• Assessments on already-active families 


Call Screening 
Supervisor weighs all 
referral and score 
information, as well as 
all recommendations, 
and approves a final 
Screening Decision. 


Screen 


Screen 

In 


Out 


Link to 
existing 
Case/GPS 
referral 


Call ^rraani ng Supervisor 

determines the Response Time 
for the referral. 


Revised business process 

(easier to read views in 
subsequent slides) 
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Process Change 


New Call Screening Business Process 

Incorporating Family Screening Score utilization and Other 8/1/16 KIDS changes 



Call Screener Findings 



Call Screener completes 
Client List, reviews referral 
information, ClientView, 
etc., and enters Initial 
Risk Rating and Initial 
Safety Rating . 


iSi 

(Z 



Call Screener is now 

able to (and must) 
generate a Family 
Screening Score . 



Call Screener enters a 
Screening 

Recommendation into 
KIDS, which is submitted 
to the Call Screening 
Supervisor . 

Recommendation is one 
of 3 choices: 


(Family Screening Score is 
inaccessible at this point.) 


Initial Risk Rating 
“locks down" at this 
point, and is no longer 
editable. 


Screen Field Screen 

Out Screen In 
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Assessing Risk at Call Screening 



Review of risk factors 

Explanation of process at call screening 



Process change 


eauaDie. 


uut 


Field Screening 

(if necessary) 


>i\.i cca ■ 


t 
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Disposition 



If Call Screening Supervisor 

determines a field screening is 
necessary, they redirect referral to 

the Field Screening Unit - 


Call Screening 
Supervisor receives 
referral and 
recommendation, and 
determines next steps. 


O 


Field Screening Findings 


Field Screening Unit 

completes a minimum of 
a Contact and a Safety 
Assessment and enters a 

Field Screening 
Recommendation into 
KIDS, which is submitted 
back to the Call 
Screening Supervisor 



If they have not 
already done so in 
determining need for 
a Field Screening. Call 
Screening Supervisor 

must generate a 
Family Screening 

Score . 


Mandatory Screen-Ins: 

If Family Screening Score is high enough to trigger a 
"Mandatory Screen-In," the Call Screening Supervisor 
must include a mandatory override justification in 
orderto Screen Out. Instances that may utilize 
override include: 

• Referrals from outside county 

• Referrals created erroneously 

• Referrals without any allegation of abuse (e.g., a 
client seeking to be connected to community 
resources) 

• Courtesy visits for other jurisdictions 

• Assessments on already-active families 


<> 


Call Screening 
Supervisor weighs all 
referral and score 
information, as well as 
all recommendations, 
and approves a final 
Screening Decision. 


Screen 


Screen 

In 


Out 


S 


Link to 
existing 
Case/GPS 
referral 


Call Screening Supervisor 

determines the Response Time 
for the referra I. 
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How the Score will look to 
Call Screening 


• The button to generate a FSS is located in two places: 

- Client details tab of call info screen 

- Outcome screen 

• The score must be generated: 

- By the screener, after entering risk and safety ratings and prior to making a recommendation 

- By a supervisor, prior to making the final decision to screen in/out 

- If additional clients were added to the referral after the score for that referral was last run 


Information 



The Screening Score needs to be reviewed since the last client was added. Please 
click the button at the bottom of the Client Details section to review. 

OK 
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How the Score will look to 
Call Screening 


• When you click the FSS button in KIDS... 


, - ,, 

No Family Screening Score was found for this referral. Click the button below to recalculate 

the Screening Score. 


Recalculate Screening Score 


L J 
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How the Score will look to 
Call Screening 


When the score is running... 


No Family Screening Score was found for this referral. Click the button below to recalculate 

the Screening Score. 


Algorithm miming... PI ease wait 


Recalculate Screening Score 
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How the Score will look to 
Call Screening 


Screening Score Historical Screening Scores 


Family Screening Score 


Recalculate Screening Score 


The purpose of the Family Screening Score is to use information 
collected by DHS and other partners to inform screening decisions. 
The Family Screening Score is calculated by integrating and 
analyzing hundreds of data elements on each person related to the 
referral to generate an overall Family Screening Score. The score 
predicts the long-term likelihood of re-referral, if the referral is 
screened out without an investigation, orhome removal, if the 
referral is screened in for investigation. 

If the Family Screening Score meets the threshold for "mandatory 
screen-in,” the call must be investigated. In all other circumstances, 
the Family Screening Score provides additional information to assist 
the Call Screening Unit in making a call screening decision and 
should not replace clinical judgement. 

The Family Screening Score is only intended to inform call screening 
decisions and is not intended to be used in making investigative or 
other child welfare decisions. 



Last Run By: 

Jane McBeth 


Last Run Date: 

4/7/2016,10:32 AM 


Algorithm Versions Used 

Re-referral v43 
Placement v22 



The display will default to the current score. You will 
click here to view previous scores for this family. 


You will hit this button to re-run the score, if needed 


This paragraph describing the purpose and use of the 
FSS will appear each time 


The score will appear on this scale, inside a blue dot. 
Along the left hand side the words "low/' "moderate," 
and "high" will appear. When the referral is a 
mandatory screen in, a new box will appear at the top 
of the scale (see subsequent slides). 


Details will appear on the bottom regarding who ran 
the score, when it was run and the version of the 
model used in the last run of the score 
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How the Score will look to 
Call Screening 


Text appearing on the thermometer: 

The purpose of the Family Screening Score is to use information collected by DHS and other partners 
to inform screening decisions. The Family Screening Score is calculated by integrating and analyzing 
hundreds of data elements on each person related to the referral to generate an overall Family 
Screening Score. The score predicts the long-term likelihood of re-referral, if the referral is screened 
out without an investigation, or home removal, if the referral is screened in for investigation. 


If the Family Screening Score meets the threshold for "mandatory screen-in," the call must be 
investigated. In all other circumstances, the Family Screening Score provides additional information to 
assist the Call Screening Unit in making a call screening decision and should not replace clinical 
judgement. 


The Family Screening Score is only intended to inform call screening decisions and is not intended to 
be used in making investigative or other child welfare decisions. 
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How the Score will look to 
Call Screening 


Screening Score Historical Screening Scores 



Layout for a score that is 
not a mandatory screen in 


Words to 

Min Score 

Max Score 

display 



Low 

1 

9 

Moderate 

10 

14 

High 

15 l 

20 
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Family Screening Score 


Recalculate Screening Score 


The purpose of the Family Screening Score is to use information 
collected by DHS and other partners to inform screening decisions. 
The Family Screening Score is calculated by integrating and 
analyzing hundreds of data elements on each person related to the 
referral to generate an overall Family Screening Score. The score 
predicts the long-term likelihood of re-referral, if the referral is 
screened out without an investigation, orhome removal, if the 
referral is screened in for investigation. 

If the Family Screening Score meets the threshold for "mandatory 
screen-in," the call must be investigated. In all other circumstances, 
the Family Screening Score provides additional information to assist 
the Call Screening Unit in making a call screening decision and 
should not replace clinical judgement. 

The Family Screening Score is only intended to inform call screening 
decisions and is not intended to be used in making investigative or 
other child welfare decisions. 



Last Run By: 

Jane McBeth 


Last Run Date: Algorithm Versions Used 

4/7/2016,10:32 AM Re-referral v43 

Placement v22 


Low Moderate 






















How the Score will look to 
Call Screening 


Layout for score that is a 
mandatory screen in 


Screening Score 


Historical Screening Scores 


x 


Family Screening Score 


Recalculate Screening Score 



The purpose of the Family screening Score is to use information 
collected by DHS and other partners to inform screening decisions. 
The Family Screening Score is calculated by integrating and 
analyzing hundreds of data elements on each person related to the 
referral togenerate an overall Family Screening Score. The score 
predicts the long-term likelihood of re-referral, if the referral is 
screened out without an investigation, or home removal, if the 
referral is screened in for investigation. 

If the Family Screening Score meets the threshold for "mandatory 
screen-in," the call must be investigated. In all other circumstances, 
the Family Screening Score provides additional information to assist 
the Call Screening Unit in making a call screening decision and 
should not replace clinical judgement. 

The Family Screening Score is only intended to inform call screening 
decisions and is not intended to be used in making investigative or 
other child welfare decisions. 


Last Run By: 

Jane McBeth 


Last Run Date: Algorithm Versions Used 

4/7/2016,10:32 AM Re-referral v43 

Placement v22 
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How the Score will look to 
Call Screening 


Layout for score that 
cannot be generated 


"Unable to recalculate score. No child 
or victim was found with an existing 
MCI ID" 


Allegheny County Department of Human Services 


Screening Score Historical Screening Scores 

Family Screening Score 

Unable to recalculate score. 

No child or victim was found with an existing MCI ID. 


Recalculate Screening Score 


The purpose of the Family Screening Score is to use information 
collected by DHS and other partners to inform screening decisions. 
The Family Screening Score is calculated by integrating and 
analyzing hundreds of data elements on each person related to the 
referral togenerate an overall Family Screening Score. The score 
predicts the long-term likelihood of re-referral, if the referral is 
screened out without an investigation, or home removal, if the 
referral is screened in for Investigation. 

If the Family Screening Score meets the threshold for "mandatory 
screen-in," the call must be investigated. In all other circumstances, 
the Family Screening Score provides additional information to assist 
the Call Screening Unit in making a call screening decision and 
should not replace clinical judgement. 

The Family Screening Score is only intended to inform call screening 
decisions and is not intended to be used in making investigative or 
other child welfare decisions. 



Last Run By: 

Jane MeBeth 


Last Run Date: 

4/7/2016, 10:32 AM 


Algorithm Versions Used 

Re-referral v43 
Placement v22 








How the Score will look to 
Call Screening 


Layout for historical scores screen 

Each previous run has its own row 
that includes: 

• Date score was run 

• Time score was run 

• Who ran the score 

• Version of algorithm used 


Screening Score Historical Screening Scores 


Historical Screening Scores 


Click on a previous Screen Score run date below to 
view the resulting Screening Score. 


Date of Run 

Time of Run 

Run By 

Algorithm Version Used 1 

1/^2016 

11:11AM 

Bruce Johnson 

Re-Referral v25 

Placement vl8 

12/20/2015 

3:21PM 

Bobby Rogers 

Re-Referral v25 

Placement vl6 

9/3/2014 

12:54 PM 

Sam Store 

Re-Referral vl4 

Placementv8 

2/22/2014 

4:43 PM 

Jimmy Slop 

Re-Referral vl3 

Placementv6 



The row highlighted (in this case, the 
third row), will display the score that 
was associated with that run along 
the right hand side 
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Low Moderate 


















Now we know what it is, we 
know what it will look like, 
we know how to run it. 

But, how does it impact our 
decision making? 

Let's look at some case 
scenarios... 



Case Scenario 1 




Case 


You said we should: 


The screening score was: 


Does that change your decision? 


1 
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Case Scenario 2 
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Case 


You said we should: 


The screening score was: 


Does that change your decision? 


2 
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Referral date: 11/2015 
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Case 


You said we should: 


The screening score was: 


Does that change your decision? 


3 
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Case Scenario 4 


• Referral date: 3/2014 
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Case 


You said we should: 


The screening score was: 


Does that change your decision? 


4 
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Case Scenario 5 



Referral date: 11/2015 




Case 


You said we should: 


The screening score was: 


Does that change your decision? 


5 
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Case Scenario 6 



45 



Case 


You said we should: 


The screening score was: 


Does that change your decision? 


6 
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Case Scenario 7 
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Case 


You said we should: 


The screening score was: 


Does that change your decision? 


7 
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How Does this Change 
the Screening Process 


in KIDS? 




Call Screening Outcome Screen 



Referral ID 

Household Name 

Referral Type 

Staff Name 

Open Date 

Closed Date 



591489 

Ikj 

CPS 


06/01/2016 




591791 

Mand 

GPS 


06/28/2016 

06/28/2016 



591397 

TEST 

GPS 


06/01/2016 




591436 

ERROR 

GPS 


06/01/2016 




591444 

Null 

GPS 


06/01/2016 




591437 

Smoke 

GPS 


06/01/2016 




591401 

Smoke 

CPS 


06/28/2016 


No changes were 


591390 

smoke 

CPS 


05/30/2016 



591389 

Smoke 

CPS 


05/31/2016 


made to this section 













of the outcome 

Case ID 

Household Name 

Case Type 

Staff Name 

Open Date 

Closed Date 

screen 



Additional History 


UP 
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Call Screeni 


Outcome 


Risk Rating* Basis for Risk Rating* 

Low Risk (L) "^1 


Safety Rating* Basis for Safety Rating* 

No safety threat v] 


Please review the Family Screening Score after entering the risk and safety ratings* 
Family Screening Score 


Call Screener Recommendation* 


Accept for Investigation v 


ng Outcome Screen 


Risk and Safety now 
follow the prior history 
section and are in a 
new section called "call 
screener findings." 

This section will be 
completed by the call 
screener. 

Risk and safety must be 
completed and 
explained before the 
FSS can be run. 

Following FSS, the 
screener will make a 
recommendation for a 
screening decision 









Call Screening Outcome Screen 


A new section called "predisposition" has been added to the outcome screen. 

This section can be used to assign referrals to active workers or to move referrals to field screening. 
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Call Screening Outcome Screen 


- Disposition- 

J Accept | v 

C : Screen Out | v 


Mandatory Screen-In Override Reason _ 

A 


V 


— Response Time*- 

O Immediate 

O Priority (Within 24 hours) 

O Children born with Substance Exposure (Within 48 hours) 
O Expedited (Within 3-7 calendar days) 

O General/Other (Within 7-10 calendar days) 


— Refer to Family and Community Teaming 

I- 

Client(s) to Contact 


v 

Reason for Referral 

P 



















Mandatory Screen-in Override 


• Mandatory screen in override 

- This can be used only by the supervisor/ROD 

- That person will select screen out, but will not be able to enter the justification until they 
attempt to save the information 

- Upon saving, they will receive a pop up indicating that they've selected to screen out a 
mandatory screen in and will have to click "ok" 

- Following that, the supervisor must enter the justification for why the referral is being screened 
out 
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Call Screening Outcome Screen 


Comments 


Supervisor Comments 


A 

V 



Assigned Staff Worker 


Program Area 




Approval Override Cancel 
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One more thing... 




Client View Dynamic Text 


• Though not a part of the Predictive Analytics changes, we are also introducing a new 
feature to KIDS at the same time as the FSS 

• The dynamic text feature will import each client on a referral's information from 
Client View into the additional information section of the call information. 

• The information included will be 

- Consistent across referrals to ensure all referrals have the same minimum Client View info 
included 

- High level to ensure that it does not replace the use of Client View when detail is needed 

• All will follow this format: 

- [Name] (Role at intake, MCI ID) has accessed the following program areas within the past 365 
days: [program areas]. There were [##] service entries, the earliest being [date] and the most 
recent being [date]. [Name] is listed as active in the following program areas: [program areas]. 
Previous history includes service access in [program areas] in [years]. 
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Client View Dynamic Text 


Critical Information Client Details 

Type and Nature of Maltreatment * 




Safety Assessment 


iP 


Child Functioning 

- 


Adult Functioning 

- 


General Parenting 

- 


Parental Discipline 

- ^\P 


After adding a 
client, their 
information 
automatically 
populates into the 
additional 
information field 



June 27, 2016 
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Client View Dynamic Text 



• Only includes information from Client View, not other resources that staff have access to (e.g., 
court records, criminal information) 


June 27, 2016 
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Questions? 




